Method and apparatus for rendering a perspective view of objects and content related thereto for location-based services on mobile device

ABSTRACT

A method including causing, at least in part, rendering of a perspective view showing one or more objects in a field of view. The method further including retrieving content associated with an object of the one or more objects in the field of view, and causing, at least in part, rendering of a graphic representation relating to the content on a surface of the object visible in the perspective view in a user interface for a location-based service of a mobile device.

BACKGROUND

Service providers (e.g., wireless, cellular, Internet, content, socialnetwork, etc.) and device manufacturers are continually challenged todeliver value and convenience to consumers by, for example, providingcompelling network services. One area of interest has been thedevelopment of mapping and navigating graphics (e.g., digital maps)and/or images (e.g., 360° panoramic street-level views of variouslocations and points of interest) augmented with, for instance,navigation tags and location relevant content. Typically, navigation,mapping, and other similar services can display either panoramic viewsor two-dimensional rendered maps. Content information is typicallylimited to use in 2D map views, and augmented reality views that attemptto display content tend to provide an unstable, cluttered display.

SOME EXAMPLE EMBODIMENTS

Therefore, there is a need for an approach for rendering a perspectiveview of objects and content related thereto for location-based serviceson a mobile device.

According to one embodiment, a method comprises causing, at least inpart, rendering of a perspective view showing one or more objects in afield of view. The method also comprises retrieving content associatedwith an object of the one or more objects in the field of view. Themethod further comprises causing, at least in part, rendering of agraphic representation relating to the content on a surface of theobject visible in the perspective view in a user interface for alocation-based service of a mobile device.

According to another embodiment, an apparatus comprising at least oneprocessor, and at least one memory including computer program code, theat least one memory and the computer program code configured to, withthe at least one processor, cause, at least in part, the apparatus tocause, at least in part, rendering of a perspective view showing one ormore objects in a field of view. The apparatus is also caused toretrieve content associated with an object of the one or more objects inthe field of view. The apparatus is further caused to cause, at least inpart, rendering of a graphic representation relating to the content on asurface of the object visible in the perspective view in a userinterface for a location-based service of the apparatus, wherein theapparatus is a mobile device.

According to another embodiment, a computer-readable storage mediumcarrying one or more sequences of one or more instructions which, whenexecuted by one or more processors, cause, at least in part, anapparatus to perform causing, at least in part, rendering of aperspective view showing one or more objects in a field of view. Theapparatus is also caused to perform retrieving content associated withan object of the one or more objects in the field of view. The apparatusis further caused to perform causing, at least in part, rendering of agraphic representation relating to the content on a surface of theobject visible in the perspective view in a user interface for alocation-based service of a mobile device.

According to another embodiment, an apparatus comprises means forcausing, at least in part, rendering of a perspective view showing oneor more objects in a field of view. The apparatus also comprises meansfor retrieving content associated with an object of the one or moreobjects in the field of view. The apparatus further comprises means forcausing, at least in part, rendering of a graphic representationrelating to the content on a surface of the object visible in theperspective view in a user interface for a location-based service of amobile device.

Still other aspects, features, and advantages of the invention arereadily apparent from the following detailed description, simply byillustrating a number of particular embodiments and implementations,including the best mode contemplated for carrying out the invention. Theinvention is also capable of other and different embodiments, and itsseveral details can be modified in various obvious respects, all withoutdeparting from the spirit and scope of the invention. Accordingly, thedrawings and description are to be regarded as illustrative in nature,and not as restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

The embodiments of the invention are illustrated by way of example, andnot by way of limitation, in the figures of the accompanying drawings:

FIG. 1 is a diagram of a system capable of rendering a perspective viewof objects and content related thereto for location-based services on amobile device, according to one embodiment;

FIG. 2 is a diagram of the components of a mapping and user interfaceapplication, according to one embodiment;

FIG. 3A is a flowchart of a process for rendering a perspective view ofobjects and content related thereto for location-based services on amobile device, according to one embodiment;

FIG. 3B is a flowchart of a process for omitting a graphicrepresentation of a distant object that is obstructed by the renderingof another object in a perspective view, according to one embodiment;

FIGS. 4A and 4B are diagrams of user interfaces utilized in theprocesses of FIGS. 3A and 3B, according to various embodiments;

FIG. 5 is a diagram of a user interface utilized in the processes ofFIGS. 3A and 3B, according to one embodiment;

FIG. 6 is a diagram of a user interface utilized in the processes ofFIGS. 3A and 3B, according to one embodiment;

FIG. 7 is a diagram of hardware that can be used to implement anembodiment of the invention;

FIG. 8 is a diagram of a chip set that can be used to implement anembodiment of the invention; and

FIG. 9 is a diagram of a mobile terminal (e.g., handset) that can beused to implement an embodiment of the invention.

DESCRIPTION OF SOME EMBODIMENTS

Examples of a method, apparatus, and computer program for rendering aperspective view of objects and content related thereto forlocation-based services on a mobile device are disclosed. In thefollowing description, for the purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of the embodiments of the invention. It is apparent,however, to one skilled in the art that the embodiments of the inventionmay be practiced without these specific details or with an equivalentarrangement. In other instances, well-known structures and devices areshown in block diagram form in order to avoid unnecessarily obscuringthe embodiments of the invention.

As used herein, the term “image” refers to one or a series of imagestaken by a camera (e.g., a still camera, digital camera, video camera,camera phone, etc.) or any other imaging equipment. Although variousembodiments are described with respect to a live camera view, it iscontemplated that the approach described herein may be used with otherlive or real-time images (e.g., a still image, a live view, a livewebcam view, etc.) as long as the image is associated with a location, atilt angle, and heading of the imaging device (e.g., camera) at the timeof image capture.

As used herein, the term “point of interest” (POI) refers to any pointspecified by a user or service provider. The term POI is also usedinterchangeably with the term “object.” By way of example, the point ofinterest can be a landmark, restaurant, museum, building, bridge, tower,dam, factory, manufacturing plant, space shuttle, etc.

As used herein, the term “perspective view” refers to any view thatprovides some perspective to an object shown therein, either when shownusing 2D or 3D displays, using or 2D or 3D images. Such perspectiveviews can be real-time images (e.g., in an augmented reality settingusing a camera of the device), a panoramic image (e.g., a pre-storedpanoramic photograph), 3D modeling in virtual reality, or other modifiedviews that attempt to show real or virtual depth to objects orsurroundings whether constructed with 2D images or 3D images.

FIG. 1 is a diagram of a system capable of rendering a perspective viewof objects and content related thereto for location-based services on amobile device, according to one embodiment.

As mentioned previously, navigation, mapping, and other like servicesand systems display either panoramic views or two-dimensional renderedmaps, however, they do not attempt to merge the two views. When contentis presented in a 2D view, certain content might be clustered or tooclose to be visible for the user. When content is presented in a 3Dview, e.g. a panoramic image or directly through camera view, then thevisible content is limited to the current scene or position of thecamera. It is usually a problem that switching views can cause confusionin the understanding of the space and location, especially when the useris not very familiar with the place in view. Some related art servicesshow content only in the map and only when the view is maximized. Otheraugmented reality or mixed reality services may display content indifferent ways depending on the kind of content; however, typically thecontent is shown in a shaking manner and not affixed in a stable mannerto an object or POI.

To address shortcomings of other related art systems, a system 100 ofFIG. 1 introduces the capability of rendering a perspective view ofobjects and content related thereto for location-based services on amobile device. The system 100 can render a user interface for alocation-based service that has a main view portion and a previewportion, which can allow a user to simultaneously visualize both aperspective view, for example, showing panoramic images of an area, anda corresponding plan view of a map of the area, and switch between suchviews as desired.

Thus, a small preview can be displayed in the mobile UI, where the mostrelevant content is shown. For example, when a user is browsing apanoramic view on the UI, the user has the option to preview the mapshowing the surroundings of what is displayed in the panorama view.Similarly, when browsing the map, the preview shows the closest panoramaof the point the user has focused on the map. Both views display thecontent that can be found in the area, allowing a better sense of thespace and location. The actions in the main view are reflected in thepreview, so that the user always has a sense of where to go physicallyif the user happens to be in the location in view or virtually if theuser is remotely browsing the area. Selecting rich content informationin a crowded area on the map can open a list view of all the content inthat crowded area, while selecting content on the panorama can open morespecific content or list view. The perspective view also limits thedisplay of graphic representations of such rich content information toobjects/POIs that are visible in the perspective view, and omits graphicrepresentations for those that are not visible, in order to provide anuncluttered perspective view.

The preview can easily be tapped to switch views and to navigate easilydepending on the user's needs. The preview can also be hidden easily bystarting a full screen view mode. If the user is navigating in the mapor plan view, the user can tap in any new location in the map and thatwill take the user's point of view to the tapped spot on the map, and atthe same time the panorama in the preview will update to the closestpanorama image from that new defined spot on the map. The user can alsorotate the phone or the point of view (POV) icon to move the orientationof the map, which will affect the orientation of the panorama preview aswell. The panorama image can be taken from the main panorama view in lowresolution to adapt in size and be quick.

This solution allows users to understand better their surroundings andthe remote surroundings when browsing location based content ornavigating in 2D maps and 3D panoramic images. The discovery of contentand the understanding of the precise place to attach content becomeeasier and nicer. Switching from one view to the other is veryintuitive, as both views show the same location and orientation.

As an example, when the user stands at a current location (e.g., theFarragut West METRO Station), the user can operate a user interface of auser device (e.g., user equipment (UE) 101) to show a plan view of a mapof the surrounding area (or of another area, such as a final destinationof the user) in a main view portion of the user interface, while aperspective view of the surrounding area is shown in a preview portionof the user interface in order to give the user an idea of the 3Dpanoramic view of the surrounding area. The perspective view can begenerated by using the camera of the user device to capture images ofthe surrounding area in real-time (e.g., in augmented reality), by usingpre-stored images (e.g., previously captured images or virtual realityimages), or a combination of real-time images and pre-stored images(e.g., mixed reality). The portion of the user interface showing theplan view of the map can include an orientation representation (e.g., aperiscope icon with an outwardly extending cone of vision) thatindicates the field of view of the perspective view. The field of visioncan be adjusted by the user by adjusting the orientation of the userdevice (e.g., utilizing a compass or other device to determine thechange in orientation), by manually manipulating the orientationrepresentation of the field of view on the plan view of the map on theuser interface, and/or by manually manipulating the view in theperspective view on the user interface. The user can switch the planview of the map from the main view portion of the user interface to thepreview portion, and thus also switch the perspective view from thepreview portion to the main view portion of the user interface. Thisdual window configuration allows a user to easily interpret the locationand orientation of the perspective view, and allows a user to quicklyand intuitively navigate to a POI or otherwise determine their location.

As noted above, the perspective view can be displayed using real-timeimages, pre-stored (pre-recorded) images, or the system 100 can retrieveand stitch a prerecorded still image right next to the live image sideby side then displays the seamlessly stitched images to the user. Tomake the switch seamlessly, the system 100 correlates a prerecordedpanoramic image that has the same tilt angle and has directional headingright next to the live image, and displays the correlated prerecordedpanoramic image on the screen. Even if two images were taken by the samedevice at the same location with the same tilt angle and the samedirectional heading, the coverage of the images can be different due toa height of the user or the settings (e.g., digital zooming, contrast,resolution, edited, clipped, etc.). If two images were taken by twodevices at the same location with the same tilt angle and the samedirectional heading, the coverage of the images can still be differentdue to different specifications of the two devices. The devices can havedifferent imaging specifications, such as LCD size, optical zoom,digital zoom, zoom wide, zoom telephoto, effective pixels, pixeldensity, image stabilization, aperture range, etc. which affect thequality and depth of images taken by two devices.

However, the existing photo matching technology allows near 100%matching between the live image and the prerecorded panoramic images.There are photo matching applications (e.g., photo-match online searchengines which compare images pixel by pixel) for choosing the bestmatched panoramic still image for the live image. There are also photostitching applications which make the boundary between the live imageand a prerecorded panoramic still image seamlessly. As the usercontinues touching the navigational arrow touching the edge of thescreen, more prerecorded panoramic still images are matched and stitchedto roll out to the screen as a panoramic view on the fly.

To navigate from the current location to a POI, the user indicates tothe system 100 the POI as the destination. By way of example, when thesystem 100 receives a target location such as the International MonetaryFund (IMF) Building as the intended POI (e.g., received as text, or on adigital map on the screen of the UE 101, etc.), the system 100 retrieveslocation data (e.g., an address, GPS coordinates, etc.) of the IMF, orthe location data of the device used to capture a prerecorded panoramicimage of the IMF (e.g., if the POI is not as well-known as the IMF, suchas a carousel in a park). The system 100 then maps a route from thecurrent location (e.g., the METRO Station) to the designated POI, andpresents the route on a digital map to the user in either the main viewportion or the preview portion. While the user is walking along theroute, the system 100 also presents a live image view of the surroundinglocation on the screen in the other of the preview portion or main viewportion. Whenever the user wants to switch among the perspective view inthe main view portion (and the plan view in the preview portion) and theplan view in the main portion (and the perspective view in the previewportion), the user can freely do so using the user interface. Otherpoints of interest may be located on the route, and a filter can be usedto select the types of POIs that are labeled using graphicrepresentations and which are not labeled.

Alternatively, the user can utilize the user interface to view a remotelocation. For example, if the user planned to visit a particular POIlater in the day, then the user could locate the POI on the plan view ofthe map (e.g., by scrolling to the location of the POI, entering anaddress of the POI, searching for the POI using keywords or the name ofthe POI, etc.), for example, in the main view portion of the userinterface. Then, the user can manipulate the orientation representationof the field of view to provide a desired vantage point. For example, ifthe user planned to travel down a certain road to get to the POI, thenthe user can manipulate the field of view to provide a vantage pointalong that road that the user will see while travelling down the roadand arriving at the POI. With the field of view set to the desiredorientation, then the user can see a preview of the perspective view ofthe POI in the preview portion of the user interface, and the user canswitch the perspective view of the POI to the main view portion of theuser interface in order to view an enlarged image of the POI. Thus, theuser will be able to see what the POI looks like, thereby allowing theuser to recognize the POI upon arrival at the POI later in the day. Theperspective view of the POI can also include graphic representations ortags (e.g., bubbles, icons, images, text, etc.) that provide a link tocontent related to the POI (e.g., name, address, telephone number,weblink, etc.), which can be selected by the user in the user interfacein order to obtain further content information regarding the POI.

In one embodiment, the system 100 displays on the screen of the UE 101different portions of the prerecorded panoramic view depending upon thetilt angle and directional heading of the UE 101 as tilted and/orrotated by the user. In this embodiment, the user can change theprerecorded panoramic image in the prerecorded panoramic view, withoutmoving/dragging a viewing tag on the screen of the UE 101.

In another embodiment, the system 100 further utilizes the augmentedreality or augmented virtuality (e.g., using 3D models and 3D mappinginformation) to insert rich content information relevant to the POI(e.g., drawn from the Internet, user inputs, etc.) in the live imageview in a real time manner. Tags are displayed on a surface of theobject or POI and virtually affixed thereto in the perspective view, andshown in a fixed 3D orientation on the surface of the object or POI. Thecontent relevant to the POI can also be seen in the prerecordedpanoramic view, and the contact may be already embedded/tagged in the inthe prerecorded panoramic view, or inserted in a real time manner. ThePOIs can be pre-set by users, service providers (e.g., wireless,cellular, Internet, content, social network, etc.), and/or devicemanufacturers, and the relevant content can be embedded/tagged by anyone of a combination of these entities as well.

By way of example, the user selects the fourth floor of a department asa POI, and tags content information of the POI retrieved from thedepartment store website. The system 100 saves the POI and the taggedcontent, and presents to the user most updated content information inthe live image view and/or the prerecorded panoramic view, automaticallyor on demand. The content information may include: (1) a floor plan ofthe POI, (2) the occupants/shops/facilities located in the POI (e.g., inthumbnail images, animation, audio alerts, etc.), (3) introduction andbackground content with respect to the occupants/shops/facilities, (4)marketing and sales content with respect to theoccupants/shops/facilities, or any other data or information tied to thePOI. It is also contemplated that content may be associated withmultiple floors. The content information includes live media, storedmedia, metadata associated with media, text information, locationinformation of other user devices, mapping data, geo-tagged data, or acombination thereof.

While the plan view of the map can show all of the graphicrepresentations for the objects, which link to the rich contentinformation thereof, in a given area, the graphic representationsaffixed to the objects in the perspective view are only shown forobjects that are visible in the field of view of the perspective view incertain embodiments. Thus, graphic representations for objects that arehidden from view in the perspective view (e.g., for objects that arehidden behind a building, or hidden behind a tree, etc.) can be omittedfrom the perspective view in order to prevent cluttering of theperspective view of the user interface.

As shown in FIG. 1, a user equipment (UE) 101 may retrieve contentinformation (e.g., content and location information) and mappinginformation (e.g., maps, GPS data, prerecorded panoramic views, etc.)from a content mapping platform 103 via a communication network 105. Thecontent and mapping information can be used by a mapping and userinterface application 107 on the UE 101 (e.g., an augmented realityapplication, navigation application, or other location-basedapplication) to a live image view and/or a prerecorded panoramic view.In the example of FIG. 1, the content mapping platform 103 storesmapping information in the map database 109 a and content information inthe content catalog 109 b. By way of example, mapping informationincludes digital maps, GPS coordinates, prerecorded panoramic views,geo-tagged data, points of interest data, or a combination thereof. Byway of example, content information includes one or more identifiers,metadata, access addresses (e.g., network address such as a UniformResource Locator (URL) or an Internet Protocol (IP) address; or a localaddress such as a file or storage location in a memory of the UE 101,description, or the like associated with content. In one embodiment,content includes live media (e.g., streaming broadcasts), stored media(e.g., stored on a network or locally), metadata associated with media,text information, location information of other user devices, or acombination thereof. The content may be provided by the service platform111 which includes one or more services 113 a-113 n (e.g., musicservice, mapping service, video service, social networking service,content broadcasting service, etc.), the one or more content providers115 a-115 m (e.g., online content retailers, public databases, etc.),other content source available or accessible over the communicationnetwork 105.

Additionally or alternatively, in certain embodiments, a user map andcontent database 117 of the UE 101 may be utilized in conjunction withthe application 107 to present content information, location information(e.g., mapping and navigation information), availability information,etc. to the user. The user may be presented with an augmented realityinterface associated with the application 107 and/or the content mappingplatform allowing 3D objects or other representations of content andrelated information to be superimposed onto an image of a physicalenvironment on the UE 101. In certain embodiments, the user interfacemay display a hybrid physical and virtual environment where 3D objectsfrom the map database 109 a are superimposed on top of a physical image.

By way of example, the UE 101 may execute the application 107 to receivecontent and/or mapping information from the content mapping platform 103or other component of the network 105. As mentioned above, the UE 101utilizes GPS satellites 119 to determine the location of the UE 101 toutilize the content mapping functions of the content mapping platform103 and/or the application 107, and the map information stored in themap database 109 a may be created from live camera views of real-worldbuildings and other sites. As such, content can be augmented intoprerecorded panoramic views and/or live camera views of real worldlocations (e.g., based on location coordinates such as globalpositioning system (GPS) coordinates).

The application 107 and the content mapping platform 103 receive accessinformation about content, determines the availability of the contentbased on the access information, and then presents a prerecordedpanoramic view or a live image view with augmented content (e.g., a livecamera view of the IMF building with augmented content, such as itsorigin, mission, facilities information: height, a number of floor,etc.). In certain embodiments, the content information may include 2Dand 3D digital maps of objects, facilities, and structures in a physicalenvironment (e.g., buildings).

By way of example, the communication network 105 of the system 100includes one or more networks such as a data network (not shown), awireless network (not shown), a telephony network (not shown), or anycombination thereof. It is contemplated that the data network may be anylocal area network (LAN), metropolitan area network (MAN), wide areanetwork (WAN), a public data network (e.g., the Internet), or any othersuitable packet-switched network, such as a commercially owned,proprietary packet-switched network, e.g., a proprietary cable orfiber-optic network. In addition, the wireless network may be, forexample, a cellular network and may employ various technologiesincluding enhanced data rates for global evolution (EDGE), generalpacket radio service (GPRS), global system for mobile communications(GSM), Internet protocol multimedia subsystem (IMS), universal mobiletelecommunications system (UMTS), etc., as well as any other suitablewireless medium, e.g., worldwide interoperability for microwave access(WiMAX), Long Term Evolution (LTE) networks, code division multipleaccess (CDMA), wideband code division multiple access (WCDMA), wirelessfidelity (WiFi), satellite, mobile ad-hoc network (MANET), and the like.

The UE 101 is any type of mobile terminal, fixed terminal, or portableterminal including a mobile handset, station, unit, device, multimediacomputer, multimedia tablet, Internet node, communicator, desktopcomputer, laptop computer, Personal Digital Assistants (PDAs), or anycombination thereof. It is also contemplated that the UE 101 can supportany type of interface to the user (such as “wearable” circuitry, etc.).

By way of example, the UE 101, and content mapping platform 103communicate with each other and other components of the communicationnetwork 105 using well known, new or still developing protocols. In thiscontext, a protocol includes a set of rules defining how the networknodes within the communication network 105 interact with each otherbased on information sent over the communication links. The protocolsare effective at different layers of operation within each node, fromgenerating and receiving physical signals of various types, to selectinga link for transferring those signals, to the format of informationindicated by those signals, to identifying which software applicationexecuting on a computer system sends or receives the information. Theconceptually different layers of protocols for exchanging informationover a network are described in the Open Systems Interconnection (OSI)Reference Model.

Communications between the network nodes are typically effected byexchanging discrete packets of data. Each packet typically comprises (1)header information associated with a particular protocol, and (2)payload information that follows the header information and containsinformation that may be processed independently of that particularprotocol. In some protocols, the packet includes (3) trailer informationfollowing the payload and indicating the end of the payload information.The header includes information such as the source of the packet, itsdestination, the length of the payload, and other properties used by theprotocol. Often, the data in the payload for the particular protocolincludes a header and payload for a different protocol associated with adifferent, higher layer of the OSI Reference Model. The header for aparticular protocol typically indicates a type for the next protocolcontained in its payload. The higher layer protocol is said to beencapsulated in the lower layer protocol. The headers included in apacket traversing multiple heterogeneous networks, such as the Internet,typically include a physical (layer 1) header, a data-link (layer 2)header, an internetwork (layer 3) header and a transport (layer 4)header, and various application headers (layer 5, layer 6 and layer 7)as defined by the OSI Reference Model.

In one embodiment, the application 107 and the content mapping platform103 may interact according to a client-server model, so that theapplication 107 of the UE 101 requests mapping and/or content data fromthe content mapping platform 103 on demand. According to theclient-server model, a client process sends a message including arequest to a server process, and the server process responds byproviding a service (e.g., providing map information). The serverprocess may also return a message with a response to the client process.Often the client process and server process execute on differentcomputer devices, called hosts, and communicate via a network using oneor more protocols for network communications. The term “server” isconventionally used to refer to the process that provides the service,or the host computer on which the process operates. Similarly, the term“client” is conventionally used to refer to the process that makes therequest, or the host computer on which the process operates. As usedherein, the terms “client” and “server” refer to the processes, ratherthan the host computers, unless otherwise clear from the context. Inaddition, the process performed by a server can be broken up to run asmultiple processes on multiple hosts (sometimes called tiers) forreasons that include reliability, scalability, and redundancy, amongothers.

FIG. 2 is a diagram of the components of a mapping and user interfaceapplication, according to one embodiment. By way of example, the mappingand user interface application 107 includes one or more components forcorrelating and navigating between a live camera image and a prerecordedpanoramic image. It is contemplated that the functions of thesecomponents may be combined in one or more components or performed byother components of equivalent functionality. In this embodiment, themapping and user interface application 107 includes at least a controllogic 201 which executes at least one algorithm for executing functionsof the mapping and user interface application 107. For example, thecontrol logic 201 interacts with an image module 203 to provide to auser a live camera view of the surrounding of a current location of theUE 101 (e.g., the Farragut West METRO Station). The image module 203 mayinclude a camera, a video camera, a combination thereof, etc. In oneembodiment, visual media is captured in the form of an image or a seriesof images.

Next, the control logic 201 interacts with a location module 205 toretrieve location data of the current location of the UE 101. In oneembodiment, the location data may include addresses, geographiccoordinates (e.g., GPS coordinates) or other indicators (e.g., longitudeand latitude information) that can be associated with the currentlocation. For example, the location data may be manually entered by theuser (e.g., entering an address or title, clicking on a digital map,etc.) or extracted or derived from any geo-tagged data. It iscontemplated that the location data or geo-tagged data could also becreated by the location module 205 by deriving the location associatedmetadata such as media titles, tags, and comments. More specifically,the location module 205 can parse the metadata for any terms thatindicate association with a particular location.

In one embodiment, the location module 205 determines the user'slocation by a triangulation system such as a GPS, assisted GPS (A-GPS)A-GPS, Cell of Origin, wireless local area network triangulation, orother location extrapolation technologies. Standard GPS and A-GPSsystems can use satellites 119 to pinpoint the location (e.g.,longitude, latitude, and altitude) of the UE 101. A Cell of Originsystem can be used to determine the cellular tower that a cellular UE101 is synchronized with. This information provides a coarse location ofthe UE 101 because the cellular tower can have a unique cellularidentifier (cell-ID) that can be geographically mapped. The locationmodule 205 may also utilize multiple technologies to detect the locationof the UE 101. GPS coordinates can provide finer detail as to thelocation of the UE 101. As previously noted, the location module 205 maybe utilized to determine location coordinates for use by the application107 and/or the content mapping platform 103.

Again, the control logic 201 interacts with the image module 203 todisplay the live camera view and location data of the current location.While displaying the live camera view of the current location, thecontrol logic 201 interacts with the image module 203 to receive anindication of switching views by the user by, for example, touching a“Switch” icon on the screen of the UE 101. The control logic 201interacts with a correlating module 207 to correlate the live image viewwith a prerecorded panoramic view with the location data, and alsointeracts with a switching module 209 to alternates/switch the displayfrom the live image view to the correlated prerecorded panoramic view.Also, the switching module 209 can control the switching between whichview (i.e., plan view or perspective view) is shown in the main viewportion of the user interface 217, and which view is shown in thepreview portion of the user interface 217.

In another embodiment, the switching module 209 interacts with amagnetometer module 211 which determines horizontal orientation ordirectional heading (e.g., a compass heading) of the UE 101, and anaccelerometer module 213 which determines vertical orientation or anangle of elevation of the UE 101. Thereafter, the switching module 209interact with the image module 203 to display on the screen of the UE101 different portions of the prerecorded panoramic view depending uponthe tilt angle and directional heading of the UE 101 as tilted and/orrotated by the user. Under these circumstances, the user can viewdifferent portions of the prerecorded panoramic view, withoutmoving/dragging a viewing tag on the screen of the UE 101.

In one embodiment, horizontal directional data obtained from amagnetometer is utilized to determine the orientation of the UE 101. Themagnetometer module 211 can include an instrument that can measure thestrength and/or direction of a magnetic field. Using the same approachas a compass, the magnetometer is capable of determining the directionalheading of a UE 101 using the magnetic field of the Earth. The front ofthe image capture device (e.g., a digital camera) (or another referencepoint on the UE 101) can be marked as a reference point in determiningdirection. Thus, if the magnetic field points north compared to thereference point, the angle the UE 101 reference point is from themagnetic field is known. Simple calculations can be made to determinethe direction of the UE 101. This directional information may becorrelated with the location information of the UE 101 to determinewhere (e.g., at which geographic feature or object) the UE 101 ispointing towards. This information may be utilized to select a portionof the prerecorded panoramic view to render to the user.

Further, the accelerometer module 213 may include an instrument that canmeasure acceleration. Using a three-axis accelerometer, with axes X, Y,and Z, provides the acceleration in three directions with known angles.Once again, the front of a media capture device can be marked as areference point in determining direction. Because the acceleration dueto gravity is known, when a UE 101 is stationary, the accelerometermodule 213 can determine the angle the UE 101 is pointed as compared toEarth's gravity. In one embodiment, vertical directional data obtainedfrom an accelerometer is used to determine the angle of elevation ortilt angle at which the UE 101 is pointing. This information inconjunction with the magnetometer information and location informationmay be utilized to determine a viewpoint in the prerecorded panoramicview to the user. As such, this information may be utilized in selectingavailable content items to present navigational information to the user.Moreover, the combined information may be utilized to determine portionsof a particular digital map or a prerecorded panoramic view that mayinterest the user.

The control logic then interacts with the image module 203 to render aviewpoint in the prerecorded panoramic view to the user, whenever theuser rotates/tilts the UE 101. As mentioned, the correlating module 207can obtain the live image from a camera and correlate the live imagewith a prerecorded panoramic view via location information. In addition,the correlating module 207 uses magnetometer information, accelerometerinformation, or a combination thereof to determine a viewpoint todisplay a portion of the prerecorded panoramic view to the user.

The control logic 201 then interacts with a content management module215 and the image module 203 to augment content information related toone or more POIs in the live image therein. The content may be receivedfrom the service platform 111, the services 113 a-113 n, the contentproviders 115 a-115 m, other like components, or a combination thereof.It is also contemplated that the user or another party authorized by theuser may manually enter a content item. In one embodiment, the contentmanagement module 215 may create a content catalog listing all contentitems and associated access addresses provided to the content managementmodule 215. In certain embodiments, the content catalog may includeadditional descriptive information and other metadata describing thecontent. The available media content or stream can take many forms(e.g., live video feeds, photographs, audio files, etc.) and can bedelivered using any number means (e.g., streaming media, downloadedmedia, spontaneously created media, etc.). The content management module215 includes one or more sub-modules or application programminginterfaces (APIs) (not pictured) for receiving and/or detecting themedia content in its native format or for converting the media contentto a media format compatible with the mapping and augmented realityapplication 107. In other embodiments, the content management module 215may initiate a download or installation of the components (e.g., codecs,player applications, etc.) needed to verify the content or stream. Thecontent data can be cached or save in the user map and content database117.

To facilitate finding specific content or features, the contentmanagement module 215 enables the user to input search criteria (e.g., acontent item, person, city, weather, etc.) and to get guidance forfinding the direction where the searched content item is located in thereal physical world. The content management module 215 also enables auser to specify a time period so as to navigate content informationusing both location and time. In one embodiment, the default time forviewing the content and mapping information is the present. If a timeperiod is set as future, the content management module 215 willdetermine the one or more content items based on the specified time,such as what will be on sales in the next three hours on the 4^(th)floor of the department store. By way of example, the sales content andproduct information can be presented on a floor plan withrepresentations of each product placed according to the associatedlocation information.

The content can be depicted as a thumbnail overlaid on the userinterface map at the location corresponding to a point of interest(e.g., a floor) or a portion of the point of interest (e.g., facilitieson the floor), and affixed to the POI at a fixed 3D orientation. Asdiscussed, the user interface may be a graphical user interface. Inaddition or alternatively, the user interface may be an audio or tactileinterface. In one embodiment, the content management nodule 215 presentsonly those content items that are available at the specified time andare not associated with any limitations or exclusive restrictions. Thisembodiment provides a user experience in which users can simply selectfrom the presented content items and be assured that the selected itemwill be available with a single selection click. In other embodiments,the content management module 215 may present all available content anddifferentiate the presentation of content available with a single clickversus content associated with additional limitations or restrictions.The specific types of presentations can be specified by the user,content provider 115, network operator, service platform 111, or acombination thereof. The content management module 215 then determineswhether to periodically update the content information.

In certain embodiments, when there is much more content available thancan be displayed in the existing user interface, the content managementnodule 215 constantly animates the display of the content items so thatnew content keeps appearing while older content disappears. Thisanimation process also makes the user interface more entertaining tousers and gives a feeling of the world being “alive” with activity.

In certain embodiments, the user map and content database 117 includesall or a portion the information in the map database 109 a and thecontent catalog 109 b. From the selected viewpoint, a live image viewaugmented with the content can be provided on the screen of the UE 101.In certain embodiments, the content management nodule 215 provides acorrelated prerecorded panoramic view from the selected view point withcontent generated or retrieved from the database 117 or the contentmapping platform 103. The content information can be embedded/tagged inthe correlated prerecorded panoramic view previously by anotherapparatus or by the content management nodule 215 on demand and/or in areal time manner when displays the correlated prerecorded panoramic viewon the screen of the UE 101.

Content and mapping information may be presented to the user via a userinterface 217, which may include various methods of communication. Forexample, the user interface 217 can have outputs including a visualcomponent (e.g., a screen), an audio component (e.g., a verbalinstructions), a physical component (e.g., vibrations), and othermethods of communication. User inputs can include a touch-screeninterface, microphone, camera, a scroll-and-click interface, a buttoninterface, etc. Further, the user may input a request to start theapplication 107 (e.g., a mapping and user interface application) andutilize the user interface 217 to receive content and mappinginformation. Through the user interface 217, the user may requestdifferent types of content, mapping, or location information to bepresented. Further, the user may be presented with 3D or augmentedreality representations of particular locations and related objects(e.g., buildings, terrain features, POIs, etc. at the particularlocation) as part of a graphical user interface on a screen of the UE101. As mentioned, the UE 101 communicates with the content mappingplatform 103 service platform 111, and/or content providers 115 a-115 mto fetch content, mapping, and or location information. The UE 101 mayutilize requests in a client server format to retrieve the content andmapping information. Moreover, the UE 101 may specify locationinformation and/or orientation information in the request to retrievethe content and mapping information.

FIG. 3A is a flowchart of a process for rendering a perspective view ofobjects and content related thereto for location-based services on amobile device, according to one embodiment. FIG. 3B is a flowchart of aprocess for omitting a graphic representation of a distant object thatis obstructed by the rendering of another object in a perspective view,according to one embodiment. FIGS. 4A, 4B, 5, and 6 are diagrams of userinterfaces utilized in the processes of FIGS. 3A and 3B, according tovarious embodiments.

In one embodiment, the mapping and user interface application 107performs the process 300 and is implemented in, for instance, a chip setincluding a processor and a memory as shown FIG. 8. In step 301, themapping and user interface application 107 causes, at least in part,rendering of a perspective view showing one or more objects in a fieldof view. for example, a user interface for a location-based service cansimultaneously include both a main view portion and a preview portion,where a perspective view is displayed in one portion (i.e., either themain view portion or the preview portion) and a plan view is displayedin another portion (i.e., the other of the preview portion or the mainview portion). Then, in step 303, the mapping and user interfaceapplication 107 retrieves content associated with an object of the oneor more objects in the field of view. Then, in step 305, the application107 causes, at least in part, rendering of a graphic representationrelating to the content on a surface of the object visible in theperspective view in the user interface for a location-based service fora mobile device. Thus, a graphic representation that relates to richcontent information relevant to the POI (e.g., drawn from the Internet,user inputs, etc.) is inserted into the perspective view, such that itis affixed to a surface of an object/POI. Thus, tags are displayed on asurface of the object or POI and virtually affixed thereto in theperspective view, and shown in a fixed 3D orientation on the surface ofthe object or POI. The content relevant to the POI can also be seen inthe prerecorded panoramic view, and the contact may be alreadyembedded/tagged in the in the prerecorded panoramic view, or inserted ina real time manner. The POIs can be pre-set by users, service providers(e.g., wireless, cellular, Internet, content, social network, etc.),and/or device manufacturers, and the relevant content can beembedded/tagged by any one of a combination of these entities as well.Thus, the user can access the rich content information by selecting thegraphic representation, which is affixed to the object/POI in a stablemanner in the perspective view.

When generating the perspective view, mapping and user interfaceapplication 107 determines an optimal image of the one or more objectsfor the perspective view from real-time images and/or pre-stored images.For example, the application 107 can compare various stored images ofthe particular field of view, and determine which image is in the bestfocus, at the best vantage point (e.g., closest vantage point to thefield of view selected by the user), and then utilize the best imageavailable. Then, the application can cause, at least in part, renderingof the perspective view in the user interface using the optimal image.Additionally, it is noted that the perspective view can be formed byproviding smooth transitions between real-time images, pre-storedimages, and/or a mix of the real-time images and the pre-stored imageswith the graphic representation overlaid onto the surface of the objectin order to provide the best possible perspective view in a smooth andcontinuous manner.

Also, it should be noted that when the application 107 receives inputchanging the orientation of the display in either the perspective viewor the plan view, then the application 107 simultaneously changes theorientation of the display of the other of perspective view and the planview in correspondence to the input. Thus, any change in one view willsimultaneously result in a change in the other view. For example, if theuser interface is showing real-time images in the perspective view andpans around in a circle, then the plan view will show a point of viewicon panning around in a circular motion as well. Also, if the user isscanning around various locations on the plan view, then the perspectiveview will be simultaneously updated to show images at the selectedlocation at the selected orientation.

In one embodiment, the mapping and user interface application 107performs the process 320 and is implemented in, for instance, a chip setincluding a processor and a memory as shown FIG. 8. In step 321, themapping and user interface application 107 determines a perspective(e.g., location, elevation, direction) of the rendering of theperspective view in relation to the one or more objects. Thus, forexample, the application 107 determines the location, elevation, anddirection at which the perspective view is shown, and compares the fieldof view of the perspective view to 3D information regarding the one ormore objects shown present in the area of the perspective view. In step323, the application 107 determines whether a rendering of a distantobject is obstructed by a rendering of another object in the perspectiveview. Thus, for example, the application 107 determines which objects inthe direction of the field of view will be visible in the perspectiveview, and which objects will not be visible since they are obstructed bycloser objects. Then, in step 325, the application 107 omits a graphicrepresentation or provides an alternative graphic representationrelating to content associated with the distant object from theperspective view when the distant object is obstructed by the renderingof another object in the perspective view. Thus, for example, when it isdetermined that a particular object in the distance will not be visiblein the perspective view, then the application 107 omits the graphicrepresentation relating to rich content information from the perspectiveview in order to keep the perspective view uncluttered. Alternatively,when it is determined that a particular object in the distance will notbe visible in the perspective view, then the application 107 can providean alternative graphic representation (e.g., a lightened or transparentimage or otherwise modified version of the original graphicrepresentation for that object) relating to rich content information inthe perspective view at a location of the hidden object/POI so that theuser can be aware of its presence but still keep the perspective viewrelatively uncluttered.

FIGS. 4A and 4B are diagrams of user interfaces utilized in theprocesses of FIGS. 3A and 3B, according to various embodiments. FIGS. 4Aand 4B depict user equipment 400, such as a mobile device, that includesa housing 401 having a display screen 403, such as a touch screen. Thedisplay screen 403 is presently displaying a user interface thatsimultaneously shows both a main view portion 405 and a preview portion407. In this embodiment, the preview portion 407 is shown as asemicircular window at a lower left-hand corner of the main viewportion; however, the preview portion can be provided in any shape andat any location on the main view portion. In the embodiment shown inFIG. 4A, the main view portion 405 is presently displaying a plan viewin which a map is shown, and the preview portion 407 is presentlydisplaying a perspective view in which a panoramic image is shown. Theplan view and the perspective view can either be displaying such viewsbased on a present location and/or orientation of the user equipment400, or based on a location selected by the user.

In the plan view, which is shown in the main view portion 405 in FIG.4A, a orientation representation is shown that includes, in thisembodiment, a periscope graphic (or also generically referred to hereinas a point of view icon) 411 and a cone shaped area 413 extending fromthe periscope graphic 411 that shows the direction in which the field ofview of the perspective view is projected and generally the area coveredby the field of view. Thus, the periscope graphic 411 and the coneshaped area 413 extending therefrom directly correlate to the field ofview shown in the perspective view. The image shown in the perspectiveview can be a real-time image captured using a camera of the UE 101, apre-stored image that is provided by a service provider and/or stored onthe UE 101, or a mixed of real-time images and pre-stored images.

In the embodiment shown in FIG. 4A, the user interface displayed on thedisplay screen 403 shows various graphic representations (e.g., abubble, as shown, or other image, icon, text, etc.) 409 that correspondto rich content information relevant to various objects/POIs. Thegraphic representations 409 that are shown in plan view can also bedisplayed in the perspective view (see, e.g., FIGS. 4B, 5, and 6) on asurface of the object/POI and virtually affixed thereto, and shown in afixed 3D orientation on the surface of the object/POI. The user canselect the graphic representation in either the plan view or in theperspective view, and a pop-up window will appear that provides the richcontent information. The content information includes live media, storedmedia, metadata associated with media, text information, locationinformation of other user devices, mapping data, geo-tagged data, or acombination thereof.

Furthermore, with respect to FIG. 4A, note that graphic representations409A, which are shown in the plan view as being in the general directionof the field of view, are omitted from the perspective view in thepreview portion 407, since the objects/POIs to which they correspond areobstructed from view and thus not visible in the perspective view.

In the embodiment shown in FIG. 4A, the user interface displayed on thedisplay screen 403 shows various icons along a bottom edge of the mainview portion 405 that provide the user with various menu options. Forexample, icon 415 can be selected by a user (e.g., using the touchscreen) to add a graphic representation for an object/POI. Thus, forexample, if the user wants to add a new graphic representation to aparticular building where a friend of the user lives, then the user canselect icon 415 and drag and drop a new graphic representation at adesired location, and then populate the graphic representation withdesired content information. Additionally, icon 417 can be selected bythe user in order to filter the types of graphic representations thatare displayed on the user interface. For example, if the user wants tomerely show graphic representations of restaurants within the plan viewand/or perspective view, then the user can select the icon 417, whichwill provide the user with a preset list of categories from which theuser can select and/or allow the user to define a desired filter orperform a text search to define the desired filter. Additionally, icon419 can be selected by the user in order to toggle between the dualwindow view (i.e., including both a main view portion and a previewportion) shown in FIGS. 4A, 4B, 4, and 6, and a full screen view mode inwhich one of either the plan view or the perspective view is shownwithout the preview portion 407.

FIG. 4B is a diagram of a user interface utilized in the processes ofFIGS. 3A and 3B, according to another embodiment. FIG. 4B depicts userequipment 400 with the display screen 403 displaying the user interfacewith the main view portion 405 showing a perspective view and thepreview portion 407 showing a corresponding plan view. In FIG. 4B, thepreview portion 407 showing the plan view includes the orientationrepresentation shown as a periscope graphic 421 and a cone shaped area423 extending from the periscope graphic 421 that shows the direction inwhich the field of view of the perspective view is projected andgenerally the area covered by the field of view. Also, in FIG. 4B, themain view portion 405 showing the perspective view includes graphicrepresentations, for example, bubble 425, which correspond to richcontent information relevant to various objects/POIs.

At any given instant of time, the user can switch which view (i.e.,perspective view or plan view) is shown in the main view portion 405 andwhich view is shown in the preview portion 407, for example, byselecting a toggle icon 427. Also, the user can zoom in and out (e.g.,simultaneously in both views, or in just one of the views, such as theview shown in the main view portion) using the plus and minus zoom icons429.

FIG. 5 is a diagram of a user interface utilized in the processes ofFIGS. 3A and 3B, according to another embodiment. FIG. 5 depicts userequipment 500, such as a mobile device, that includes a housing 501having a display screen 503. The display screen 503 is presentlydisplaying a user interface that simultaneously shows both a main viewportion 505 and a preview portion 507. In the embodiment shown in FIG.5, the main view portion 505 is presently displaying a plan view inwhich a map is shown, and the preview portion 507 is presentlydisplaying a perspective view in which a panoramic image is shown. Theplan view and the perspective view can either be displaying such viewsbased on a present location and/or orientation of the user equipment500, or based on a location selected by the user.

In FIG. 5, the main view portion 505 showing the plan view includes theorientation representation shown as a periscope graphic 509 and a coneshaped area 511 extending from the periscope graphic 509 that shows thedirection in which the field of view of the perspective view isprojected and generally the area covered by the field of view. Also, themain view portion 505 includes graphic representations, for example,bubble 513, which correspond to rich content information relevant to therespective object/POI. Furthermore, the preview portion 507 includesgraphic representations, such as bubble 515, which corresponds to thesame object/POI as bubble 513 in the plan view. The graphicrepresentation 515 displayed in the perspective view is shown on asurface of the object/POI and virtually affixed thereto, and shown in afixed 3D orientation on the surface of the object/POI. For example, thegraphic representation 515 is affixed to a most prominent surface of theobject/POI, as viewed in the perspective view or as preset by the useror service provider. The graphical representation can be a bubble icon,an image (e.g., set by the user, such as a picture of a person that livesin the building, or set by the service provider), icon that isrepresentative of the category of the object/POI (e.g., a fork and knifeindicating a restaurant POI, a shopping cart represent a store POI,etc.), text (e.g., name or description of the object/POI), etc.

Furthermore, with respect to FIG. 5, note that graphic representation513A, which is shown in the plan view as being in the general directionof the field of view, is omitted from the perspective view in thepreview portion 507, since the object/POI to which it corresponds isobstructed from view and thus not visible in the perspective view. Thus,the perspective view maintains an uncluttered appearance.

FIG. 6 is a diagram of a user interface utilized in the processes ofFIGS. 3A and 3B, according to another embodiment. FIG. 6 depicts userequipment 600 that includes a housing 601 having a display screen 603.The display screen 603 is presently displaying a user interface thatsimultaneously shows both a main view portion 605 and a preview portion607. In the embodiment shown in FIG. 6, the main view portion 605 ispresently displaying a perspective view in which a panoramic image isshown, and the preview portion 607 is presently displaying a plan viewin which a map is shown. The plan view and the perspective view caneither be displaying such views based on a present location and/ororientation of the user equipment 600, or based on a location selectedby the user.

In FIG. 6, the preview portion 607 showing the plan view includes theorientation representation shown as a periscope graphic 609 and a coneshaped area 611 extending from the periscope graphic 609 that shows thedirection in which the field of view of the perspective view isprojected and generally the area covered by the field of view. Also, themain view portion 605 includes graphic representations, for example,bubble 613, which correspond to rich content information relevant to therespective object/POI. Furthermore, the main view portion 605 includesother graphic representations, for example bubble 615, which correspondto rich content information relevant to other objects/POIs that arevisible in the field of view of the perspective view. While the planview of the map can show all of the graphic representations for theobjects, which link to the rich content information thereof, in a givenarea, the graphic representations affixed to the objects in theperspective view are only shown for objects that are visible in thefield of view of the perspective view in certain embodiments. Thus,graphic representations for objects that are hidden from view in theperspective view (e.g., for objects that are hidden behind a building,or hidden behind a tree, etc.) are omitted from the perspective view inorder to prevent cluttering of the perspective view of the userinterface.

The processes described herein for rendering a perspective view ofobjects and content related thereto for location-based services on amobile device may be advantageously implemented via software, hardware,firmware or a combination of software and/or firmware and/or hardware.For example, the processes described herein, including for providinguser interface navigation information associated with the availabilityof services, may be advantageously implemented via processor(s), DigitalSignal Processing (DSP) chip, an Application Specific Integrated Circuit(ASIC), Field Programmable Gate Arrays (FPGAs), etc. Such exemplaryhardware for performing the described functions is detailed below.

FIG. 7 illustrates a computer system 700 upon which an embodiment of theinvention may be implemented. Although computer system 700 is depictedwith respect to a particular device or equipment, it is contemplatedthat other devices or equipment (e.g., network elements, servers, etc.)within FIG. 7 can deploy the illustrated hardware and components ofsystem 700. Computer system 700 is programmed (e.g., via computerprogram code or instructions) to render a perspective view of objectsand content related thereto for location-based services on a mobiledevice as described herein and includes a communication mechanism suchas a bus 710 for passing information between other internal and externalcomponents of the computer system 700. Information (also called data) isrepresented as a physical expression of a measurable phenomenon,typically electric voltages, but including, in other embodiments, suchphenomena as magnetic, electromagnetic, pressure, chemical, biological,molecular, atomic, sub-atomic and quantum interactions. For example,north and south magnetic fields, or a zero and non-zero electricvoltage, represent two states (0, 1) of a binary digit (bit). Otherphenomena can represent digits of a higher base. A superposition ofmultiple simultaneous quantum states before measurement represents aquantum bit (qubit). A sequence of one or more digits constitutesdigital data that is used to represent a number or code for a character.In some embodiments, information called analog data is represented by anear continuum of measurable values within a particular range. Computersystem 700, or a portion thereof, constitutes a means for performing oneor more steps of rendering a perspective view of objects and contentrelated thereto for location-based services on a mobile device.

A bus 710 includes one or more parallel conductors of information sothat information is transferred quickly among devices coupled to the bus710. One or more processors 702 for processing information are coupledwith the bus 710.

A processor (or multiple processors) 702 performs a set of operations oninformation as specified by computer program code related to render aperspective view of objects and content related thereto forlocation-based services on a mobile device. The computer program code isa set of instructions or statements providing instructions for theoperation of the processor and/or the computer system to performspecified functions. The code, for example, may be written in a computerprogramming language that is compiled into a native instruction set ofthe processor. The code may also be written directly using the nativeinstruction set (e.g., machine language). The set of operations includebringing information in from the bus 710 and placing information on thebus 710. The set of operations also typically include comparing two ormore units of information, shifting positions of units of information,and combining two or more units of information, such as by addition ormultiplication or logical operations like OR, exclusive OR (XOR), andAND. Each operation of the set of operations that can be performed bythe processor is represented to the processor by information calledinstructions, such as an operation code of one or more digits. Asequence of operations to be executed by the processor 702, such as asequence of operation codes, constitute processor instructions, alsocalled computer system instructions or, simply, computer instructions.Processors may be implemented as mechanical, electrical, magnetic,optical, chemical or quantum components, among others, alone or incombination.

Computer system 700 also includes a memory 704 coupled to bus 710. Thememory 704, such as a random access memory (RAM) or other dynamicstorage device, stores information including processor instructions forrendering a perspective view of objects and content related thereto forlocation-based services on a mobile device. Dynamic memory allowsinformation stored therein to be changed by the computer system 700. RAMallows a unit of information stored at a location called a memoryaddress to be stored and retrieved independently of information atneighboring addresses. The memory 704 is also used by the processor 702to store temporary values during execution of processor instructions.The computer system 700 also includes a read only memory (ROM) 706 orother static storage device coupled to the bus 710 for storing staticinformation, including instructions, that is not changed by the computersystem 700. Some memory is composed of volatile storage that loses theinformation stored thereon when power is lost. Also coupled to bus 710is a non-volatile (persistent) storage device 708, such as a magneticdisk, optical disk or flash card, for storing information, includinginstructions, that persists even when the computer system 700 is turnedoff or otherwise loses power.

Information, including instructions for rendering a perspective view ofobjects and content related thereto for location-based services on amobile device, is provided to the bus 710 for use by the processor froman external input device 712, such as a keyboard containing alphanumerickeys operated by a human user, or a sensor. A sensor detects conditionsin its vicinity and transforms those detections into physical expressioncompatible with the measurable phenomenon used to represent informationin computer system 700. Other external devices coupled to bus 710, usedprimarily for interacting with humans, include a display device 714,such as a cathode ray tube (CRT) or a liquid crystal display (LCD), orplasma screen or printer for presenting text or images, and a pointingdevice 716, such as a mouse or a trackball or cursor direction keys, ormotion sensor, for controlling a position of a small cursor imagepresented on the display 714 and issuing commands associated withgraphical elements presented on the display 714. In some embodiments,for example, in embodiments in which the computer system 700 performsall functions automatically without human input, one or more of externalinput device 712, display device 714 and pointing device 716 is omitted.

In the illustrated embodiment, special purpose hardware, such as anapplication specific integrated circuit (ASIC) 720, is coupled to bus710. The special purpose hardware is configured to perform operationsnot performed by processor 702 quickly enough for special purposes.Examples of application specific ICs include graphics accelerator cardsfor generating images for display 714, cryptographic boards forencrypting and decrypting messages sent over a network, speechrecognition, and interfaces to special external devices, such as roboticarms and medical scanning equipment that repeatedly perform some complexsequence of operations that are more efficiently implemented inhardware.

Computer system 700 also includes one or more instances of acommunications interface 770 coupled to bus 710. Communication interface770 provides a one-way or two-way communication coupling to a variety ofexternal devices that operate with their own processors, such asprinters, scanners and external disks. In general the coupling is with anetwork link 778 that is connected to a local network 780 to which avariety of external devices with their own processors are connected. Forexample, communication interface 770 may be a parallel port or a serialport or a universal serial bus (USB) port on a personal computer. Insome embodiments, communications interface 770 is an integrated servicesdigital network (ISDN) card or a digital subscriber line (DSL) card or atelephone modem that provides an information communication connection toa corresponding type of telephone line. In some embodiments, acommunication interface 770 is a cable modem that converts signals onbus 710 into signals for a communication connection over a coaxial cableor into optical signals for a communication connection over a fiberoptic cable. As another example, communications interface 770 may be alocal area network (LAN) card to provide a data communication connectionto a compatible LAN, such as Ethernet. Wireless links may also beimplemented. For wireless links, the communications interface 770 sendsor receives or both sends and receives electrical, acoustic orelectromagnetic signals, including infrared and optical signals, thatcarry information streams, such as digital data. For example, inwireless handheld devices, such as mobile telephones like cell phones,the communications interface 770 includes a radio band electromagnetictransmitter and receiver called a radio transceiver. In certainembodiments, the communications interface 770 enables connection to thecommunication network 105 for rendering a perspective view of objectsand content related thereto for location-based services on a mobiledevice to the UE 101.

The term “computer-readable medium” as used herein refers to any mediumthat participates in providing information to processor 702, includinginstructions for execution. Such a medium may take many forms,including, but not limited to computer-readable storage medium (e.g.,non-volatile media, volatile media), and transmission media.Non-transitory media, such as non-volatile media, include, for example,optical or magnetic disks, such as storage device 708. Volatile mediainclude, for example, dynamic memory 704. Transmission media include,for example, coaxial cables, copper wire, fiber optic cables, andcarrier waves that travel through space without wires or cables, such asacoustic waves and electromagnetic waves, including radio, optical andinfrared waves. Signals include man-made transient variations inamplitude, frequency, phase, polarization or other physical propertiestransmitted through the transmission media. Common forms ofcomputer-readable media include, for example, a floppy disk, a flexibledisk, hard disk, magnetic tape, any other magnetic medium, a CD-ROM,CDRW, DVD, any other optical medium, punch cards, paper tape, opticalmark sheets, any other physical medium with patterns of holes or otheroptically recognizable indicia, a RAM, a PROM, an EPROM, a FLASH-EPROM,any other memory chip or cartridge, a carrier wave, or any other mediumfrom which a computer can read. The term computer-readable storagemedium is used herein to refer to any computer-readable medium excepttransmission media.

Logic encoded in one or more tangible media includes one or both ofprocessor instructions on a computer-readable storage media and specialpurpose hardware, such as ASIC 720.

Network link 778 typically provides information communication usingtransmission media through one or more networks to other devices thatuse or process the information. For example, network link 778 mayprovide a connection through local network 780 to a host computer 782 orto equipment 784 operated by an Internet Service Provider (ISP). ISPequipment 784 in turn provides data communication services through thepublic, world-wide packet-switching communication network of networksnow commonly referred to as the Internet 790.

A computer called a server host 792 connected to the Internet hosts aprocess that provides a service in response to information received overthe Internet. For example, server host 792 hosts a process that providesinformation representing video data for presentation at display 714. Itis contemplated that the components of system 700 can be deployed invarious configurations within other computer systems, e.g., host 782 andserver 792.

At least some embodiments of the invention are related to the use ofcomputer system 700 for implementing some or all of the techniquesdescribed herein. According to one embodiment of the invention, thosetechniques are performed by computer system 700 in response to processor702 executing one or more sequences of one or more processorinstructions contained in memory 704. Such instructions, also calledcomputer instructions, software and program code, may be read intomemory 704 from another computer-readable medium such as storage device708 or network link 778. Execution of the sequences of instructionscontained in memory 704 causes processor 702 to perform one or more ofthe method steps described herein. In alternative embodiments, hardware,such as ASIC 720, may be used in place of or in combination withsoftware to implement the invention. Thus, embodiments of the inventionare not limited to any specific combination of hardware and software,unless otherwise explicitly stated herein.

The signals transmitted over network link 778 and other networks throughcommunications interface 770, carry information to and from computersystem 700. Computer system 700 can send and receive information,including program code, through the networks 780, 790 among others,through network link 778 and communications interface 770. In an exampleusing the Internet 790, a server host 792 transmits program code for aparticular application, requested by a message sent from computer 700,through Internet 790, ISP equipment 784, local network 780 andcommunications interface 770. The received code may be executed byprocessor 702 as it is received, or may be stored in memory 704 or instorage device 708 or other non-volatile storage for later execution, orboth. In this manner, computer system 700 may obtain application programcode in the form of signals on a carrier wave.

Various forms of computer readable media may be involved in carrying oneor more sequence of instructions or data or both to processor 702 forexecution. For example, instructions and data may initially be carriedon a magnetic disk of a remote computer such as host 782. The remotecomputer loads the instructions and data into its dynamic memory andsends the instructions and data over a telephone line using a modem. Amodem local to the computer system 700 receives the instructions anddata on a telephone line and uses an infra-red transmitter to convertthe instructions and data to a signal on an infra-red carrier waveserving as the network link 778. An infrared detector serving ascommunications interface 770 receives the instructions and data carriedin the infrared signal and places information representing theinstructions and data onto bus 710. Bus 710 carries the information tomemory 704 from which processor 702 retrieves and executes theinstructions using some of the data sent with the instructions. Theinstructions and data received in memory 704 may optionally be stored onstorage device 708, either before or after execution by the processor702.

FIG. 8 illustrates a chip set or chip 800 upon which an embodiment ofthe invention may be implemented. Chip set 800 is programmed to render aperspective view of objects and content related thereto forlocation-based services on a mobile device as described herein andincludes, for instance, the processor and memory components describedwith respect to FIG. 7 incorporated in one or more physical packages(e.g., chips). By way of example, a physical package includes anarrangement of one or more materials, components, and/or wires on astructural assembly (e.g., a baseboard) to provide one or morecharacteristics such as physical strength, conservation of size, and/orlimitation of electrical interaction. It is contemplated that in certainembodiments the chip set 800 can be implemented in a single chip. It isfurther contemplated that in certain embodiments the chip set or chip800 can be implemented as a single “system on a chip.” It is furthercontemplated that in certain embodiments a separate ASIC would not beused, for example, and that all relevant functions as disclosed hereinwould be performed by a processor or processors. Chip set or chip 800,or a portion thereof, constitutes a means for performing one or moresteps of providing user interface navigation information associated withthe availability of services. Chip set or chip 800, or a portionthereof, constitutes a means for performing one or more steps ofrendering a perspective view of objects and content related thereto forlocation-based services on a mobile device.

In one embodiment, the chip set or chip 800 includes a communicationmechanism such as a bus 801 for passing information among the componentsof the chip set 800. A processor 803 has connectivity to the bus 801 toexecute instructions and process information stored in, for example, amemory 805. The processor 803 may include one or more processing coreswith each core configured to perform independently. A multi-coreprocessor enables multiprocessing within a single physical package.Examples of a multi-core processor include two, four, eight, or greaternumbers of processing cores. Alternatively or in addition, the processor803 may include one or more microprocessors configured in tandem via thebus 801 to enable independent execution of instructions, pipelining, andmultithreading. The processor 803 may also be accompanied with one ormore specialized components to perform certain processing functions andtasks such as one or more digital signal processors (DSP) 807, or one ormore application-specific integrated circuits (ASIC) 809. A DSP 807typically is configured to process real-world signals (e.g., sound) inreal time independently of the processor 803. Similarly, an ASIC 809 canbe configured to performed specialized functions not easily performed bya more general purpose processor. Other specialized components to aid inperforming the inventive functions described herein may include one ormore field programmable gate arrays (FPGA) (not shown), one or morecontrollers (not shown), or one or more other special-purpose computerchips.

In one embodiment, the chip set or chip 800 includes merely one or moreprocessors and some software and/or firmware supporting and/or relatingto and/or for the one or more processors.

The processor 803 and accompanying components have connectivity to thememory 805 via the bus 801. The memory 805 includes both dynamic memory(e.g., RAM, magnetic disk, writable optical disk, etc.) and staticmemory (e.g., ROM, CD-ROM, etc.) for storing executable instructionsthat when executed perform the inventive steps described herein torender a perspective view of objects and content related thereto forlocation-based services on a mobile device. The memory 805 also storesthe data associated with or generated by the execution of the inventivesteps.

FIG. 9 is a diagram of exemplary components of a mobile terminal (e.g.,handset) for communications, which is capable of operating in the systemof FIG. 1, according to one embodiment. In some embodiments, mobileterminal 900, or a portion thereof, constitutes a means for performingone or more steps of rendering a perspective view of objects and contentrelated thereto for location-based services on a mobile device.Generally, a radio receiver is often defined in terms of front-end andback-end characteristics. The front-end of the receiver encompasses allof the Radio Frequency (RF) circuitry whereas the back-end encompassesall of the base-band processing circuitry. As used in this application,the term “circuitry” refers to both: (1) hardware-only implementations(such as implementations in only analog and/or digital circuitry), and(2) to combinations of circuitry and software (and/or firmware) (suchas, if applicable to the particular context, to a combination ofprocessor(s), including digital signal processor(s), software, andmemory(ies) that work together to cause an apparatus, such as a mobilephone or server, to perform various functions). This definition of“circuitry” applies to all uses of this term in this application,including in any claims. As a further example, as used in thisapplication and if applicable to the particular context, the term“circuitry” would also cover an implementation of merely a processor (ormultiple processors) and its (or their) accompanying software/orfirmware. The term “circuitry” would also cover if applicable to theparticular context, for example, a baseband integrated circuit orapplications processor integrated circuit in a mobile phone or a similarintegrated circuit in a cellular network device or other networkdevices.

Pertinent internal components of the telephone include a Main ControlUnit (MCU) 903, a Digital Signal Processor (DSP) 905, and areceiver/transmitter unit including a microphone gain control unit and aspeaker gain control unit. A main display unit 907 provides a display tothe user in support of various applications and mobile terminalfunctions that perform or support the steps of rendering a perspectiveview of objects and content related thereto for location-based serviceson a mobile device. The display 9 includes display circuitry configuredto display at least a portion of a user interface of the mobile terminal(e.g., mobile telephone). Additionally, the display 907 and displaycircuitry are configured to facilitate user control of at least somefunctions of the mobile terminal. An audio function circuitry 909includes a microphone 911 and microphone amplifier that amplifies thespeech signal output from the microphone 911. The amplified speechsignal output from the microphone 911 is fed to a coder/decoder (CODEC)913.

A radio section 915 amplifies power and converts frequency in order tocommunicate with a base station, which is included in a mobilecommunication system, via antenna 917. The power amplifier (PA) 919 andthe transmitter/modulation circuitry are operationally responsive to theMCU 903, with an output from the PA 919 coupled to the duplexer 921 orcirculator or antenna switch, as known in the art. The PA 919 alsocouples to a battery interface and power control unit 920.

In use, a user of mobile terminal 901 speaks into the microphone 911 andhis or her voice along with any detected background noise is convertedinto an analog voltage. The analog voltage is then converted into adigital signal through the Analog to Digital Converter (ADC) 923. Thecontrol unit 903 routes the digital signal into the DSP 905 forprocessing therein, such as speech encoding, channel encoding,encrypting, and interleaving. In one embodiment, the processed voicesignals are encoded, by units not separately shown, using a cellulartransmission protocol such as global evolution (EDGE), general packetradio service (GPRS), global system for mobile communications (GSM),Internet protocol multimedia subsystem (IMS), universal mobiletelecommunications system (UMTS), etc., as well as any other suitablewireless medium, e.g., microwave access (WiMAX), Long Term Evolution(LTE) networks, code division multiple access (CDMA), wideband codedivision multiple access (WCDMA), wireless fidelity (WiFi), satellite,and the like.

The encoded signals are then routed to an equalizer 925 for compensationof any frequency-dependent impairments that occur during transmissionthough the air such as phase and amplitude distortion. After equalizingthe bit stream, the modulator 927 combines the signal with a RF signalgenerated in the RF interface 929. The modulator 927 generates a sinewave by way of frequency or phase modulation. In order to prepare thesignal for transmission, an up-converter 931 combines the sine waveoutput from the modulator 927 with another sine wave generated by asynthesizer 933 to achieve the desired frequency of transmission. Thesignal is then sent through a PA 919 to increase the signal to anappropriate power level. In practical systems, the PA 919 acts as avariable gain amplifier whose gain is controlled by the DSP 905 frominformation received from a network base station. The signal is thenfiltered within the duplexer 921 and optionally sent to an antennacoupler 935 to match impedances to provide maximum power transfer.Finally, the signal is transmitted via antenna 917 to a local basestation. An automatic gain control (AGC) can be supplied to control thegain of the final stages of the receiver. The signals may be forwardedfrom there to a remote telephone which may be another cellulartelephone, other mobile phone or a land-line connected to a PublicSwitched Telephone Network (PSTN), or other telephony networks.

Voice signals transmitted to the mobile terminal 901 are received viaantenna 917 and immediately amplified by a low noise amplifier (LNA)937. A down-converter 939 lowers the carrier frequency while thedemodulator 941 strips away the RF leaving only a digital bit stream.The signal then goes through the equalizer 925 and is processed by theDSP 905. A Digital to Analog Converter (DAC) 943 converts the signal andthe resulting output is transmitted to the user through the speaker 945,all under control of a Main Control Unit (MCU) 903—which can beimplemented as a Central Processing Unit (CPU) (not shown).

The MCU 903 receives various signals including input signals from thekeyboard 947. The keyboard 947 and/or the MCU 903 in combination withother user input components (e.g., the microphone 911) comprise a userinterface circuitry for managing user input. The MCU 903 runs a userinterface software to facilitate user control of at least some functionsof the mobile terminal 901 to render a perspective view of objects andcontent related thereto for location-based services on a mobile device.The MCU 903 also delivers a display command and a switch command to thedisplay 907 and to the speech output switching controller, respectively.Further, the MCU 903 exchanges information with the DSP 905 and canaccess an optionally incorporated SIM card 949 and a memory 951. Inaddition, the MCU 903 executes various control functions required of theterminal. The DSP 905 may, depending upon the implementation, performany of a variety of conventional digital processing functions on thevoice signals. Additionally, DSP 905 determines the background noiselevel of the local environment from the signals detected by microphone911 and sets the gain of microphone 911 to a level selected tocompensate for the natural tendency of the user of the mobile terminal901.

The CODEC 913 includes the ADC 923 and DAC 943. The memory 951 storesvarious data including call incoming tone data and is capable of storingother data including music data received via, e.g., the global Internet.The software module could reside in RAM memory, flash memory, registers,or any other form of writable storage medium known in the art. Thememory device 951 may be, but not limited to, a single memory, CD, DVD,ROM, RAM, EEPROM, optical storage, or any other non-volatile storagemedium capable of storing digital data.

An optionally incorporated SIM card 949 carries, for instance, importantinformation, such as the cellular phone number, the carrier supplyingservice, subscription details, and security information. The SIM card949 serves primarily to identify the mobile terminal 901 on a radionetwork. The card 949 also contains a memory for storing a personaltelephone number registry, text messages, and user specific mobileterminal settings.

While the invention has been described in connection with a number ofembodiments and implementations, the invention is not so limited butcovers various obvious modifications and equivalent arrangements, whichfall within the purview of the appended claims. Although features of theinvention are expressed in certain combinations among the claims, it iscontemplated that these features can be arranged in any combination andorder.

1. A method comprising: causing, at least in part, rendering of aperspective view showing one or more objects in a field of view;retrieving content associated with an object of the one or more objectsin the field of view; and causing, at least in part, rendering of agraphic representation relating to the content on a surface of theobject visible in the perspective view in a user interface for alocation-based service of a mobile device.
 2. A method of claim 1,wherein the rendering of the graphic representation includes orientingthe graphic representation with respect to the surface of the objectvisible in the perspective view.
 3. A method of claim 1, furthercomprising: determining a perspective of the rendering of theperspective view in relation to the one or more objects; determiningwhether a rendering of a distant object is obstructed by a rendering ofanother object in the perspective view; and omitting a graphicrepresentation or providing an alternative graphic representationrelating to content associated with the distant object from theperspective view when the distant object is obstructed by the renderingof another object in the perspective view.
 4. A method of claim 1,further comprising: determining an optimal image of the one or moreobjects for the perspective view from real-time images and/or pre-storedimages; and causing, at least in part, rendering of the perspective viewin the user interface using the optimal image.
 5. A method of claim 1,further comprising: receiving an input for selecting the graphicrepresentation via the user interface; and causing, at least in part,display of the content associated with the object on the user interface.6. A method of claim 1, further comprising: receiving real-time imagescaptured by the mobile device of the one or more objects in the field ofview, wherein the perspective view shows the real-time images with thegraphic representation overlaid onto the surface of the object.
 7. Amethod of claim 1, further comprising: receiving pre-stored images ofthe one or more objects in the field of view, wherein the perspectiveview shows the pre-stored images with the graphic representationoverlaid onto the surface of the object.
 8. A method of claim 1, furthercomprising: receiving real-time images captured by the mobile device ofthe one or more objects in the field of view; and receiving pre-storedimages of the one or more objects in the field of view, wherein theperspective view shows a mix of the real-time images and the pre-storedimages with the graphic representation overlaid onto the surface of theobject.
 9. A method of claim 1, further comprising: receiving real-timeimages captured by the mobile device of the one or more objects in thefield of view; and receiving pre-stored images of the one or moreobjects in the field of view, wherein the perspective view transitionsbetween the real-time images, the pre-stored images, and a mix of thereal-time images and the pre-stored images with the graphicrepresentation overlaid onto the surface of the object.
 10. An apparatuscomprising: at least one processor; and at least one memory includingcomputer program code for one or more programs, the at least one memoryand the computer program code configured to, with the at least oneprocessor, cause the apparatus to perform at least the following, cause,at least in part, rendering of a perspective view showing one or moreobjects in a field of view; retrieve content associated with an objectof the one or more objects in the field of view; and cause, at least inpart, rendering of a graphic representation relating to the content on asurface of the object visible in the perspective view in a userinterface for a location-based service of the apparatus, wherein theapparatus is a mobile device.
 11. An apparatus of claim 10, wherein therendering of the graphic representation includes orienting the graphicrepresentation with respect to the surface of the object visible in theperspective view.
 12. An apparatus of claim 10, wherein the apparatus isfurther caused to: determine a perspective of the rendering of theperspective view in relation to the one or more objects; determinewhether a rendering of a distant object is obstructed by a rendering ofanother object in the perspective view; and omit a graphicrepresentation or provide an alternative graphic representation relatingto content associated with the distant object from the perspective viewwhen the distant object is obstructed by the rendering of another objectin the perspective view.
 13. An apparatus of claim 10, wherein theapparatus is further caused to: determine an optimal image of the one ormore objects for the perspective view from real-time images and/orpre-stored images; and cause, at least in part, rendering of theperspective view in the user interface using the optimal image.
 14. Anapparatus of claim 10, wherein the apparatus is further caused to:receive an input for selecting the graphic representation via the userinterface; and cause, at least in part, display of the contentassociated with the object on the user interface.
 15. An apparatus ofclaim 10, wherein the apparatus is further caused to: receive real-timeimages captured by the mobile device of the one or more objects in thefield of view, wherein the perspective view shows the real-time imageswith the graphic representation overlaid onto the surface of the object.16. An apparatus of claim 10, wherein the apparatus is further causedto: receive pre-stored images of the one or more objects in the field ofview, wherein the perspective view shows the pre-stored images with thegraphic representation overlaid onto the surface of the object.
 17. Anapparatus of claim 10, wherein the apparatus is further caused to:receive real-time images captured by the mobile device of the one ormore objects in the field of view; and receive pre-stored images of theone or more objects in the field of view, wherein the perspective viewshows a mix of the real-time images and the pre-stored images with thegraphic representation overlaid onto the surface of the object.
 18. Anapparatus of claim 10, wherein the apparatus is further caused to:receive real-time images captured by the mobile device of the one ormore objects in the field of view; and receive pre-stored images of theone or more objects in the field of view, wherein the perspective viewtransitions between the real-time images, the pre-stored images, and amix of the real-time images and the pre-stored images with the graphicrepresentation overlaid onto the surface of the object.
 19. Acomputer-readable storage medium carrying one or more sequences of oneor more instructions which, when executed by one or more processors,cause an apparatus to at least perform the following steps: causing, atleast in part, rendering of a perspective view showing one or moreobjects in a field of view; retrieving content associated with an objectof the one or more objects in the field of view; and causing, at leastin part, rendering of a graphic representation relating to the contenton a surface of the object visible in the perspective view in a userinterface for a location-based service of a mobile device.
 20. Acomputer-readable storage medium of claim 19, wherein the rendering ofthe graphic representation includes orienting the graphic representationwith respect to the surface of the object visible in the perspectiveview.